{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|default_exp models.TST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an unofficial PyTorch implementation by Ignacio Oguiza of  - oguiza@timeseriesAI.co based on:\n",
    "* George Zerveas et al. A Transformer-based Framework for Multivariate Time Series Representation Learning, in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '21), August 14--18, 2021. ArXiV version: https://arxiv.org/abs/2010.02803\n",
    "* Official implementation: https://github.com/gzerveas/mvts_transformer\n",
    "\n",
    "```bash\n",
    "@inproceedings{10.1145/3447548.3467401,\n",
    "author = {Zerveas, George and Jayaraman, Srideepika and Patel, Dhaval and Bhamidipaty, Anuradha and Eickhoff, Carsten},\n",
    "title = {A Transformer-Based Framework for Multivariate Time Series Representation Learning},\n",
    "year = {2021},\n",
    "isbn = {9781450383325},\n",
    "publisher = {Association for Computing Machinery},\n",
    "address = {New York, NY, USA},\n",
    "url = {https://doi.org/10.1145/3447548.3467401},\n",
    "doi = {10.1145/3447548.3467401},\n",
    "booktitle = {Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &amp; Data Mining},\n",
    "pages = {2114–2124},\n",
    "numpages = {11},\n",
    "keywords = {regression, framework, multivariate time series, classification, transformer, deep learning, self-supervised learning, unsupervised learning, imputation},\n",
    "location = {Virtual Event, Singapore},\n",
    "series = {KDD '21}\n",
    "}\n",
    "```\n",
    "\n",
    "\n",
    "This paper uses 'Attention is all you need' as a major reference:\n",
    "* Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). **Attention is all you need**. In Advances in neural information processing systems (pp. 5998-6008).\n",
    "\n",
    "This implementation is adapted to work with the rest of the `tsai` library, and contain some hyperparameters that are not available in the original implementation. They are included to experiment with them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TST arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usual values are the ones that appear in the \"Attention is all you need\" and \"A Transformer-based Framework for Multivariate Time Series Representation Learning\" papers. \n",
    "\n",
    "The default values are the ones selected as a default configuration in the latter.\n",
    "\n",
    "* c_in: the number of features (aka variables, dimensions, channels) in the time series dataset. dls.var\n",
    "* c_out: the number of target classes. dls.c\n",
    "* seq_len: number of time steps in the time series. dls.len\n",
    "* max_seq_len: useful to control the temporal resolution in long time series to avoid memory issues. Default. None.\n",
    "* d_model: total dimension of the model (number of features created by the model). Usual values: 128-1024. Default: 128.\n",
    "* n_heads:  parallel attention heads. Usual values: 8-16. Default: 16.\n",
    "* d_k: size of the learned linear projection of queries and keys in the MHA. Usual values: 16-512. Default: None -> (d_model/n_heads) = 32.\n",
    "* d_v: size of the learned linear projection of values in the MHA. Usual values: 16-512. Default: None -> (d_model/n_heads) = 32.\n",
    "* d_ff: the dimension of the feedforward network model. Usual values: 256-4096. Default: 256.\n",
    "* dropout: amount of residual dropout applied in the encoder. Usual values: 0.-0.3. Default: 0.1.\n",
    "* activation: the activation function of intermediate layer, relu or gelu. Default: 'gelu'.\n",
    "* n_layers: the number of sub-encoder-layers in the encoder. Usual values: 2-8. Default: 3.\n",
    "* fc_dropout: dropout applied to the final fully connected layer. Usual values: 0.-0.8. Default: 0.\n",
    "* y_range: range of possible y values (used in regression tasks). Default: None\n",
    "* kwargs: nn.Conv1d kwargs. If not {}, a nn.Conv1d with those kwargs will be applied to original time series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "from tsai.imports import *\n",
    "from tsai.utils import *\n",
    "from tsai.models.layers import *\n",
    "from tsai.models.utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|exporti\n",
    "class _ScaledDotProductAttention(Module):\n",
    "    def __init__(self, d_k:int): self.d_k = d_k\n",
    "    def forward(self, q:Tensor, k:Tensor, v:Tensor, mask:Optional[Tensor]=None):\n",
    "\n",
    "        # MatMul (q, k) - similarity scores for all pairs of positions in an input sequence\n",
    "        scores = torch.matmul(q, k)                                         # scores : [bs x n_heads x q_len x q_len]\n",
    "        \n",
    "        # Scale\n",
    "        scores = scores / (self.d_k ** 0.5)\n",
    "        \n",
    "        # Mask (optional)\n",
    "        if mask is not None: scores.masked_fill_(mask, -1e9)\n",
    "        \n",
    "        # SoftMax\n",
    "        attn = F.softmax(scores, dim=-1)                                    # attn   : [bs x n_heads x q_len x q_len]\n",
    "        \n",
    "        # MatMul (attn, v)\n",
    "        context = torch.matmul(attn, v)                                     # context: [bs x n_heads x q_len x d_v]\n",
    "        \n",
    "        return context, attn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|exporti\n",
    "class _MultiHeadAttention(Module):\n",
    "    def __init__(self, d_model:int, n_heads:int, d_k:int, d_v:int):\n",
    "        r\"\"\"\n",
    "        Input shape:  Q, K, V:[batch_size (bs) x q_len x d_model], mask:[q_len x q_len]\n",
    "        \"\"\"\n",
    "        self.n_heads, self.d_k, self.d_v = n_heads, d_k, d_v\n",
    "        \n",
    "        self.W_Q = nn.Linear(d_model, d_k * n_heads, bias=False)\n",
    "        self.W_K = nn.Linear(d_model, d_k * n_heads, bias=False)\n",
    "        self.W_V = nn.Linear(d_model, d_v * n_heads, bias=False)\n",
    "        \n",
    "        self.W_O = nn.Linear(n_heads * d_v, d_model, bias=False)\n",
    "\n",
    "    def forward(self, Q:Tensor, K:Tensor, V:Tensor, mask:Optional[Tensor]=None):\n",
    "        \n",
    "        bs = Q.size(0)\n",
    "\n",
    "        # Linear (+ split in multiple heads)\n",
    "        q_s = self.W_Q(Q).view(bs, -1, self.n_heads, self.d_k).transpose(1,2)       # q_s    : [bs x n_heads x q_len x d_k]\n",
    "        k_s = self.W_K(K).view(bs, -1, self.n_heads, self.d_k).permute(0,2,3,1)     # k_s    : [bs x n_heads x d_k x q_len] - transpose(1,2) + transpose(2,3)\n",
    "        v_s = self.W_V(V).view(bs, -1, self.n_heads, self.d_v).transpose(1,2)       # v_s    : [bs x n_heads x q_len x d_v]\n",
    "\n",
    "        # Scaled Dot-Product Attention (multiple heads)\n",
    "        context, attn = _ScaledDotProductAttention(self.d_k)(q_s, k_s, v_s)          # context: [bs x n_heads x q_len x d_v], attn: [bs x n_heads x q_len x q_len]\n",
    "\n",
    "        # Concat\n",
    "        context = context.transpose(1, 2).contiguous().view(bs, -1, self.n_heads * self.d_v) # context: [bs x q_len x n_heads * d_v]\n",
    "\n",
    "        # Linear\n",
    "        output = self.W_O(context)                                                  # context: [bs x q_len x d_model]\n",
    "        \n",
    "        return output, attn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([16, 50, 128]), torch.Size([16, 3, 50, 50]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = torch.rand(16, 50, 128)\n",
    "output, attn = _MultiHeadAttention(d_model=128, n_heads=3, d_k=8, d_v=6)(t, t, t)\n",
    "output.shape, attn.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|exporti\n",
    "def get_activation_fn(activation):\n",
    "    if activation == \"relu\": return nn.ReLU()\n",
    "    elif activation == \"gelu\": return nn.GELU()\n",
    "    else: return activation()\n",
    "#         raise ValueError(f'{activation} is not available. You can use \"relu\" or \"gelu\"')\n",
    "\n",
    "class _TSTEncoderLayer(Module):\n",
    "    def __init__(self, q_len:int, d_model:int, n_heads:int, d_k:Optional[int]=None, d_v:Optional[int]=None, d_ff:int=256, dropout:float=0.1, \n",
    "                 activation:str=\"gelu\"):\n",
    "\n",
    "        assert d_model // n_heads, f\"d_model ({d_model}) must be divisible by n_heads ({n_heads})\"\n",
    "        d_k = ifnone(d_k, d_model // n_heads)\n",
    "        d_v = ifnone(d_v, d_model // n_heads)\n",
    "\n",
    "        # Multi-Head attention\n",
    "        self.self_attn = _MultiHeadAttention(d_model, n_heads, d_k, d_v)\n",
    "\n",
    "        # Add & Norm\n",
    "        self.dropout_attn = nn.Dropout(dropout)\n",
    "        self.batchnorm_attn = nn.Sequential(Transpose(1,2), nn.BatchNorm1d(d_model), Transpose(1,2))\n",
    "\n",
    "        # Position-wise Feed-Forward\n",
    "        self.ff = nn.Sequential(nn.Linear(d_model, d_ff), \n",
    "                                get_activation_fn(activation), \n",
    "                                nn.Dropout(dropout), \n",
    "                                nn.Linear(d_ff, d_model))\n",
    "\n",
    "        # Add & Norm\n",
    "        self.dropout_ffn = nn.Dropout(dropout)\n",
    "        self.batchnorm_ffn = nn.Sequential(Transpose(1,2), nn.BatchNorm1d(d_model), Transpose(1,2))\n",
    "\n",
    "    def forward(self, src:Tensor, mask:Optional[Tensor]=None) -> Tensor:\n",
    "\n",
    "        # Multi-Head attention sublayer\n",
    "        ## Multi-Head attention\n",
    "        src2, attn = self.self_attn(src, src, src, mask=mask)\n",
    "        ## Add & Norm\n",
    "        src = src + self.dropout_attn(src2) # Add: residual connection with residual dropout\n",
    "        src = self.batchnorm_attn(src)      # Norm: batchnorm \n",
    "\n",
    "        # Feed-forward sublayer\n",
    "        ## Position-wise Feed-Forward\n",
    "        src2 = self.ff(src)\n",
    "        ## Add & Norm\n",
    "        src = src + self.dropout_ffn(src2) # Add: residual connection with residual dropout\n",
    "        src = self.batchnorm_ffn(src) # Norm: batchnorm\n",
    "\n",
    "        return src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([16, 50, 128])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = torch.rand(16, 50, 128)\n",
    "output = _TSTEncoderLayer(q_len=50, d_model=128, n_heads=3, d_k=None, d_v=None, d_ff=512, dropout=0.1, activation='gelu')(t)\n",
    "output.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|exporti\n",
    "class _TSTEncoder(Module):\n",
    "    def __init__(self, q_len, d_model, n_heads, d_k=None, d_v=None, d_ff=None, dropout=0.1, activation='gelu', n_layers=1):\n",
    "        \n",
    "        self.layers = nn.ModuleList([_TSTEncoderLayer(q_len, d_model, n_heads=n_heads, d_k=d_k, d_v=d_v, d_ff=d_ff, dropout=dropout, \n",
    "                                                            activation=activation) for i in range(n_layers)])\n",
    "\n",
    "    def forward(self, src):\n",
    "        output = src\n",
    "        for mod in self.layers: output = mod(output)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "class TST(Module):\n",
    "    def __init__(self, c_in:int, c_out:int, seq_len:int, max_seq_len:Optional[int]=None, \n",
    "                 n_layers:int=3, d_model:int=128, n_heads:int=16, d_k:Optional[int]=None, d_v:Optional[int]=None,  \n",
    "                 d_ff:int=256, dropout:float=0.1, act:str=\"gelu\", fc_dropout:float=0., \n",
    "                 y_range:Optional[tuple]=None, verbose:bool=False, **kwargs):\n",
    "        r\"\"\"TST (Time Series Transformer) is a Transformer that takes continuous time series as inputs.\n",
    "        As mentioned in the paper, the input must be standardized by_var based on the entire training set.\n",
    "        Args:\n",
    "            c_in: the number of features (aka variables, dimensions, channels) in the time series dataset.\n",
    "            c_out: the number of target classes.\n",
    "            seq_len: number of time steps in the time series.\n",
    "            max_seq_len: useful to control the temporal resolution in long time series to avoid memory issues.\n",
    "            d_model: total dimension of the model (number of features created by the model)\n",
    "            n_heads:  parallel attention heads.\n",
    "            d_k: size of the learned linear projection of queries and keys in the MHA. Usual values: 16-512. Default: None -> (d_model/n_heads) = 32.\n",
    "            d_v: size of the learned linear projection of values in the MHA. Usual values: 16-512. Default: None -> (d_model/n_heads) = 32.\n",
    "            d_ff: the dimension of the feedforward network model.\n",
    "            dropout: amount of residual dropout applied in the encoder.\n",
    "            act: the activation function of intermediate layer, relu or gelu.\n",
    "            n_layers: the number of sub-encoder-layers in the encoder.\n",
    "            fc_dropout: dropout applied to the final fully connected layer.\n",
    "            y_range: range of possible y values (used in regression tasks).\n",
    "            kwargs: nn.Conv1d kwargs. If not {}, a nn.Conv1d with those kwargs will be applied to original time series.\n",
    "\n",
    "        Input shape:\n",
    "            bs (batch size) x nvars (aka features, variables, dimensions, channels) x seq_len (aka time steps)\n",
    "        \"\"\"\n",
    "        self.c_out, self.seq_len = c_out, seq_len\n",
    "        \n",
    "        # Input encoding\n",
    "        q_len = seq_len\n",
    "        self.new_q_len = False\n",
    "        if max_seq_len is not None and seq_len > max_seq_len: # Control temporal resolution\n",
    "            self.new_q_len = True\n",
    "            q_len = max_seq_len\n",
    "            tr_factor = math.ceil(seq_len / q_len)\n",
    "            total_padding = (tr_factor * q_len - seq_len)\n",
    "            padding = (total_padding // 2, total_padding - total_padding // 2)\n",
    "            self.W_P = nn.Sequential(Pad1d(padding), Conv1d(c_in, d_model, kernel_size=tr_factor, padding=0, stride=tr_factor))\n",
    "            pv(f'temporal resolution modified: {seq_len} --> {q_len} time steps: kernel_size={tr_factor}, stride={tr_factor}, padding={padding}.\\n', verbose)\n",
    "        elif kwargs:\n",
    "            self.new_q_len = True\n",
    "            t = torch.rand(1, 1, seq_len)\n",
    "            q_len = nn.Conv1d(1, 1, **kwargs)(t).shape[-1]\n",
    "            self.W_P = nn.Conv1d(c_in, d_model, **kwargs) # Eq 2\n",
    "            pv(f'Conv1d with kwargs={kwargs} applied to input to create input encodings\\n', verbose)\n",
    "        else:\n",
    "            self.W_P = nn.Linear(c_in, d_model) # Eq 1: projection of feature vectors onto a d-dim vector space\n",
    "\n",
    "        # Positional encoding\n",
    "        W_pos = torch.empty((q_len, d_model), device=default_device())\n",
    "        nn.init.uniform_(W_pos, -0.02, 0.02)\n",
    "        self.W_pos = nn.Parameter(W_pos, requires_grad=True)\n",
    "\n",
    "        # Residual dropout\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "        # Encoder\n",
    "        self.encoder = _TSTEncoder(q_len, d_model, n_heads, d_k=d_k, d_v=d_v, d_ff=d_ff, dropout=dropout, activation=act, n_layers=n_layers)\n",
    "        self.flatten = Flatten()\n",
    "        \n",
    "        # Head\n",
    "        self.head_nf = q_len * d_model\n",
    "        self.head = self.create_head(self.head_nf, c_out, act=act, fc_dropout=fc_dropout, y_range=y_range)\n",
    "\n",
    "    def create_head(self, nf, c_out, act=\"gelu\", fc_dropout=0., y_range=None, **kwargs):\n",
    "        layers = [get_activation_fn(act), Flatten()]\n",
    "        if fc_dropout: layers += [nn.Dropout(fc_dropout)]\n",
    "        layers += [nn.Linear(nf, c_out)]\n",
    "        if y_range: layers += [SigmoidRange(*y_range)]\n",
    "        return nn.Sequential(*layers)    \n",
    "        \n",
    "\n",
    "    def forward(self, x:Tensor, mask:Optional[Tensor]=None) -> Tensor:  # x: [bs x nvars x q_len]\n",
    "\n",
    "        # Input encoding\n",
    "        if self.new_q_len: u = self.W_P(x).transpose(2,1) # Eq 2        # u: [bs x d_model x q_len] transposed to [bs x q_len x d_model]\n",
    "        else: u = self.W_P(x.transpose(2,1)) # Eq 1                     # u: [bs x q_len x nvars] converted to [bs x q_len x d_model]\n",
    "\n",
    "        # Positional encoding\n",
    "        u = self.dropout(u + self.W_pos)\n",
    "\n",
    "        # Encoder\n",
    "        z = self.encoder(u)                                             # z: [bs x q_len x d_model]\n",
    "        z = z.transpose(2,1).contiguous()                               # z: [bs x d_model x q_len]\n",
    "\n",
    "        # Classification/ Regression head\n",
    "        return self.head(z)                                             # output: [bs x c_out]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model parameters: 517378\n"
     ]
    }
   ],
   "source": [
    "bs = 32\n",
    "c_in = 9  # aka channels, features, variables, dimensions\n",
    "c_out = 2\n",
    "seq_len = 5000\n",
    "\n",
    "xb = torch.randn(bs, c_in, seq_len)\n",
    "\n",
    "# standardize by channel by_var based on the training set\n",
    "xb = (xb - xb.mean((0, 2), keepdim=True)) / xb.std((0, 2), keepdim=True)\n",
    "\n",
    "# Settings\n",
    "max_seq_len = 256\n",
    "d_model = 128\n",
    "n_heads = 16\n",
    "d_k = d_v = None # if None --> d_model // n_heads\n",
    "d_ff = 256\n",
    "dropout = 0.1\n",
    "activation = \"gelu\"\n",
    "n_layers = 3\n",
    "fc_dropout = 0.1\n",
    "kwargs = {}\n",
    "\n",
    "model = TST(c_in, c_out, seq_len, max_seq_len=max_seq_len, d_model=d_model, n_heads=n_heads,\n",
    "            d_k=d_k, d_v=d_v, d_ff=d_ff, dropout=dropout, activation=activation, n_layers=n_layers,\n",
    "            fc_dropout=fc_dropout, **kwargs)\n",
    "test_eq(model.to(xb.device)(xb).shape, [bs, c_out])\n",
    "print(f'model parameters: {count_parameters(model)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model parameters: 420226\n"
     ]
    }
   ],
   "source": [
    "bs = 32\n",
    "c_in = 9  # aka channels, features, variables, dimensions\n",
    "c_out = 2\n",
    "seq_len = 60\n",
    "\n",
    "xb = torch.randn(bs, c_in, seq_len)\n",
    "\n",
    "# standardize by channel by_var based on the training set\n",
    "xb = (xb - xb.mean((0, 2), keepdim=True)) / xb.std((0, 2), keepdim=True)\n",
    "\n",
    "# Settings\n",
    "max_seq_len = 120\n",
    "d_model = 128\n",
    "n_heads = 16\n",
    "d_k = d_v = None # if None --> d_model // n_heads\n",
    "d_ff = 256\n",
    "dropout = 0.1\n",
    "act = \"gelu\"\n",
    "n_layers = 3\n",
    "fc_dropout = 0.1\n",
    "kwargs = {}\n",
    "# kwargs = dict(kernel_size=5, padding=2)\n",
    "\n",
    "model = TST(c_in, c_out, seq_len, max_seq_len=max_seq_len, d_model=d_model, n_heads=n_heads,\n",
    "            d_k=d_k, d_v=d_v, d_ff=d_ff, dropout=dropout, act=act, n_layers=n_layers,\n",
    "            fc_dropout=fc_dropout, **kwargs)\n",
    "test_eq(model.to(xb.device)(xb).shape, [bs, c_out])\n",
    "print(f'model parameters: {count_parameters(model)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": "IPython.notebook.save_checkpoint();",
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/nacho/notebooks/tsai/nbs/108b_models.TST.ipynb saved at 2022-11-09 13:05:31\n",
      "Correct notebook to script conversion! 😃\n",
      "Wednesday 09/11/22 13:05:34 CET\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" autoplay=\"autoplay\">\n",
       "                    <source src=\"data:audio/wav;base64,UklGRvQHAABXQVZFZm10IBAAAAABAAEAECcAACBOAAACABAAZGF0YdAHAAAAAPF/iPh/gOoOon6w6ayCoR2ZeyfbjobxK+F2Hs0XjKc5i3DGvzaTlEaraE+zz5uLUl9f46fHpWJdxVSrnfmw8mYEScqUP70cb0Q8X41uysJ1si6Eh1jYzXp9IE2DzOYsftYRyoCY9dJ/8QICgIcEun8D9PmAaBPlfT7lq4MFIlh61tYPiCswIHX+yBaOqT1QbuW7qpVQSv9lu6+xnvRVSlyopAypbGBTUdSalrSTaUBFYpInwUpxOzhti5TOdndyKhCGrdwAfBUcXIJB69p+Vw1egB76+n9q/h6ADglbf4LvnIHfF/981ODThF4m8HiS0riJVjQ6c+/EOZCYQfJrGrhBmPVNMmNArLKhQlkXWYqhbaxXY8ZNHphLuBJsZUEckCTFVHMgNKGJytIDeSUmw4QN4Qx9pReTgb3vYX/TCBuApf75f+P5Y4CRDdN+B+tngk8c8nt03CKGqipgd13OhotwOC5x9MCAknFFcmlmtPmagFFFYOCo0qRzXMhVi57pryNmIEqJlRi8bm52PfuNM8k4dfQv+4cO12l6zCGdg3jl730uE/KAPvS+f0wEAoAsA89/XfXQgBESIn6S5luDtiC8eh/YmIfpLqt1OMp5jXg8/24MveqUNUnPZsqw0Z3yVDldnaUOqIZfXlKrm36zzWhjRhaT+r+ncHI5/otUzfd2uSt7hl/bqXtoHaCC6+mqfrAOeoDD+PJ/xf8RgLMHfH/b8GeBihZIfSXidoQSJWB52NM1iRkzz3MkxpKPbUCrbDu5d5fgTAxkSK3JoEhYD1p2omere2LZTuqYLbdWa49Cx5Dww7tyXDUnioXRkHhwJyKFvd/AfPoYy4Fl7j1/LQorgEr9/X89+0qAOAwAf13sJoL8Gkd8wt25hWIp3Heez/eKODfPcSPCzpFNRDVqf7UlmnNQKGHgqd+jgVvJVm2f265QZTpLS5byur1tpT6ajvrHq3Q2MXWIxtUCehoj8YMk5LB9hRQegeTypn+nBQWA0QHgf7f2q4C5EFt+5ucOg2YfHXtq2SSHpS0ydnTL4IxFO6pvNb4ulBdInWfcsfSc7VMmXpSmE6eeXmZThJxpsgRohEfOk86+AHCoOpOMFsx1dv8s6oYT2k17uR7ngpXod34IEJqAaPfnfyABCIBZBpl/NPI2gTQVjX134x2ExSPMeR7VtYjZMWJ0W8ftjkA/YW1durCWykvjZFKu4p9LVwVbZKNkqpxh6U+6mRC2mGq2Q3SRvsIgcpc2sIpD0Bp4uiiFhW3ecXxOGgaCDe0Vf4cLPoDv+/5/mfw1gN4KKX+17emBqBmYfBHfVYUZKFR44NBtiv41bHJUwx+RJkP1apu2VJlkTwli4qrwoo1ax1dToNCtemRSTBGXz7kJbdM/PY/Dxht0dTLziH7Ul3loJEiE0uJsfdsVTYGL8Yt/AgcMgHYA7X8S+IqAYA+QfjzpxIIVHnp7tdqzhmAstXaxzEqMETpScGC/dJP3Rmdo8LIZnOVSEF+Opxumsl1sVF+dVrE5Z6NIiZSkvVdv2zsqjdnK8HVDLlyHyNjuegogM4NA5z9+YRG9gA722H97AgOA/gSyf43zCIHdE899yuTIg3ciNXpm1jmImTDwdJPITI4RPhRugbvslbFKt2Vfr/6eTFb4W1WkY6m6YPdQjJr2tNZp3EQlko7BgXHRNz2LAc+gdwMq7IUf3R58ohtFgrbr6n7hDFWAlPr8f/T9I4CECU9/De+vgVQY5nxh4POEzybJeCTS5YnCNAZzhsRzkP1Bsmu4t4aYU07nYuerA6KWWcJYO6HHrKJjaE3Zl624UWz/QOOPjcWHc7QzdIk40yl5tCWjhIDhJX0xF4CBMvBsf10IF4Ac//Z/bPlsgAcOwn6S6n6CwxzUewLcRoYaKzV38M23i9o493CNwL6S1UUuaQe0QpvbUfdfiqglpcRccFU+nkWwambASUiVfLyqbg49xY2eyWh1hy/Sh37XjHpaIYKD7OUEfrgS5IC09MV/1gMBgKMDyH/n9N6AhhINfh7mdoMoIZt6r9fAh1cvfHXNya6N4DzDbqi8K5WWSYlmbbAdnkpV6FxJpWSo1V8DUmGb3rMRaQBG2JJgwN9wCDnNi8HNI3dKK1aG0dvHe/UciIJf6rt+Og5wgDn59X9P/xWAKQhxf2XweYH+FjB9suGVhIMlOnlo02GJhTOdc7vFyo/TQGxs2Li7lz9NwmPurBihnVi7WSWiwKvGYntOpJiOt5drKUKMkFnE8HLxNPmJ9NG4eP8mAYUv4Np8hhi3gdruSX+3CSWAwP38f8f6UoCuDPF+6Os8gnAbKnxQ3d2F0imydzDPKIuiN5lxu8EKkrFE82kftW2az1DbYImpMqTUW3FWIJ83r5hl2koJlla7+m0+PmSOZcjcdMgwS4g11iZ6qCLUg5jkxn0QFA6BWvOvfzEFBIBHAtp/Qfa3gC4RSH5y5yeD2B/8evnYS4cULgR2CMsUja47cG/QvW6UeEhXZ3+xP51GVNVdP6Zpp+1eDFM5nMeySWghR4+TNL85cD46YIyCzKJ2kCzEhoTabXtGHs+CCemJfpMPjoDe9+t/qQALgM8Gj3++8UaBqRV2fQTjO4Q3JKd5r9TgiEYyMHTxxiWPpz8jbfq585YpTJpk960xoKFXsVoTo7yq6GGMTw==\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#|eval: false\n",
    "#|hide\n",
    "from tsai.export import get_nb_name; nb_name = get_nb_name(locals())\n",
    "from tsai.imports import create_scripts; create_scripts(nb_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
